Yield prediction for a cornfield

ABSTRACT

A method for predicting the yield of a cornfield is presented. The method has the steps of positioning a digital camera at a defined distance above an average plane of ears of a cornfield, capturing a digital image of a section of the cornfield, determining the area of the captured cornfield section, determining the total area of the ears in the digital image compared to the total area of the digital image, and determining the yield of the field from the total area of the ears in the digital image compared to the total area of the digital image, the determined area of the captured cornfield section, the average grain weight of an ear, a total field area and a calibration factor.

The invention generally relates to prediction of a yield of a harvestand in particular to a method for predicting the yield of a cornfield.The invention further relates to a corresponding system for predictingthe yield of a cornfield and to a computer system relating thereto.

Now that automation of classic production areas in industrialmanufacturing is well advanced, this trend is now continuing in theclassic agricultural field. Although the use of monitoring technology isnot yet the universal standard in agricultural production processes,there is progress in this area as well. In many cases, classic industry4.0 technologies can also be applied to agricultural processes andmachines. In this case, however, at least one factor must be taken intoconsideration: the selected technologies should be easy to manage androbust. Moreover, the technologies used must be economical, as pricepressure is also continuously mounting in production plants due toglobalization.

Now and then, on the other hand, there are still reservations withrespect to the use of high technology on farms, as dealing with thesetechnologies is not part of standard knowledge in this field.Accordingly, information technology to be used in agriculturalbusinesses must allow simple and intuitive operation.

Depending on the country, there are indeed several regions of the worldin which highly sophisticated smart farming or digital farminginitiatives are already in place, some of which have also achieved ahigh degree of automation in field work; with relatively little expense,however, it is possible to achieve considerable improvements if analysistechniques supported by image processing and information technology aremade available in simple form and on a broad scale. This can beimplemented most favorably in the form of technologies that have alreadypenetrated into the daily life of the population.

Few quantitative parameters have been used to date in predicting yieldsin agriculture—in particular in predicting yields of cornfields. In mostcases, these are still empirical values. In addition to reliable weatherdata, further analysis techniques would also be helpful in allowingfamers to make highly precise predictions with respect to their cropyields in order to decide the best time for harvesting. In this way, itwould be possible for producers to take advantage of customer purchaseprice fluctuations-such as e.g. those of agricultural cooperatives orindustrial bulk purchasers—in order to optimize profits.

Accordingly, there is a need for improved yield prediction in cornfieldsand other agriculturally used surfaces. The subject matter of thepresent document addresses this objective.

The object of this application is achieved by means of the subjectmatter of the independent claims. Further examples are given in therespective dependent claims.

According to a first aspect of the present invention, a method ispresented for predicting the yield of a cornfield. The method cancomprise positioning a digital camera at a defined distance above anaverage plane of ears of a cornfield and capturing a digital image of asection of the cornfield with the positioned digital camera. The averageplane of the ears can lie parallel to an image plane of the digitalcamera.

Furthermore, the method can comprise determining the area of thecaptured cornfield section from the defined distance and a viewing angleof the digital camera and determining the total area of the ears in thedigital image compared to the total area of the digital image by meansof an algorithm for differentiating between image pixels of the ears andother image pixels that do not belong to ears.

In addition, the method can comprise determining the yield of the fieldfrom the total area of the ears in the digital image. This can becarried out compared to the total area of the digital image, thedetermined area of the captured cornfield section, the average grainweight of an ear, a total field area and a first calibration factor.

According to a further aspect of the invention, a system for predictingthe yield of a cornfield is presented. The system can comprise a digitalcamera that is positioned at a defined distance above an average planeof ears of a cornfield. The digital camera can be adapted for capturinga digital image of a section of the cornfield with the positioneddigital camera. The average plane of the ears and an image plane of thedigital camera can lie parallel to each other.

Additionally, the system can comprise a partial area determination unitfor determining the area of the captured cornfield section from thedefined distance and a viewing angle of the digital camera and an eararea determination unit that is adapted for determining the total areaof the ears in the digital image. This can be carried out compared tothe total area of the digital image by means of an algorithm fordifferentiating between image pixels of the ears and other image pixelsthat do not belong to the ears.

Furthermore, the system can have a yield determination module fordetermining the yield of the field from the total area of the ears inthe digital image compared to the total area of the digital image, thedetermined area of the captured cornfield section, the average grainweight of an ear, a total field area and a calibration factor.

It should be noted that the system presented can be implemented as partof a smartphone. Moreover, by means of a high-performance form of asmartphone, the method presented here can also be completely orpartially carried out by said smartphone. Alternatively, determining thetotal number of grains of an ear can also be carried out on a dedicatedcomputer specially adapted for this purpose, a server computer or anyother computer system.

For this purpose, it can be necessary for embodiments to be in the formof a corresponding computer program product. This product can compriseinstructions which, when executed on a computer system, carry out stepsof the method described.

The method presented for determining a weight of all of the grains of anear of a grain stalk and the corresponding system have a series ofadvantages and technical effects:

Because of the simplicity of the means used for determining a yield—orfor yield prediction—of a cornfield, it is possible to use the methodanywhere without major expense. In cases where the digital images arefirst sent to a computer center for evaluation, this computer center canbe located at virtually any desired location. The calculation can beoffered as a service. On the other hand, if the evaluation is carriedout directly in an application (“app”) on a smartphone that alsocomprises the digital camera, there are no additional communicationexpenses, and the evaluation results are available virtuallyimmediately. In addition to a smartphone, the digital camera, as well asthe necessary evaluation logic, and optionally communication units formobile data communication can be located in another device, or in adevice designed specially for the purpose provided according to theinvention.

The alternative possible positionings of the digital camera above thecornfield make it possible on the one hand to produce the digital imageof an elevated location above the surface of the cornfield (e.g. from anagricultural machine); on the other hand, it is also possible to producethe required digital image while standing in the cornfield using thedigital camera mounted on a rod.

In addition, the proposed method for yield prediction can be combinedwith a further elegant method for determining the grain weight of anear. On the one hand, this makes it possible to estimate the yield of acornfield directly from images of the ears of the cornfield. The samedigital camera can be used in the method for yield analysis and in thepartial method for determining the grain weight. Moreover, thecalculations for yield prediction and the calculations for determiningthe grain weight can be carried out on the same computer system,allowing the results of the one calculation (grain weight) to be madeavailable for the second calculation (yield prediction) as input values.

If a high-performance smartphone or another dedicated device havingcorrespondingly high-performance electronics and a digital camera isavailable, the above-mentioned determinations (grain weight and yieldprediction) can be carried out directly on this smartphone or the otherdedicated device.

As mentioned above, for advance determination of the grain weight of anear, the same digital camera—e.g. in a smartphone—can be used as a frontend for the improved yield prediction. The use of a mobile device-suchas a mobile telephone with a camera—is sufficient to allow farmers tomake an improved yield prediction of their cornfield. For this purpose,it can be important as a point of departure to determine the yieldpotential of an ear of a cornfield. One or two digital images of an earare sufficient to allow the farmer to make a significantly improvedprediction of the yield of a field. The use of a further everyday itemin the form of a reference card does not make the management oracceptance of the method more complex.

In this process, the ear can either be picked or cut off and placed onthe reference card, or the ear can remain on the stalk and the referencecard can simply be placed behind the ear. The scale on the referencecard provides a clear and genuine measuring criterion together with thedigital image.

The computing power required for automated measurement of the ear andthe grain weight of the ear can be provided by a computer center. Thiscomputer center-just like the computer center for calculating the fieldyield—can be operated at any desired location. A computer can be used bya farmer, a plurality of farmers can operate the computer together, or aservice provider can take over the analysis work and provide therequired computing power. The latter possibility is further advantageousin that this service could be provided in the form of a cloud computingservice for a large number of farmers in different regions or also inmultiple countries. This would also make it easier to take into accountparallels among different regions, global as well as local weatherinfluences, or regionally known pest infestation, use of fertilizer, useof insecticides, etc.

The digital image can be transmitted via a mobile network to anevaluation computer. The analysis can be carried out, and the result canbe wirelessly transmitted back to the farmer or the mobile device. Bymeans of further methods, one could extrapolate from the grain weight ofan ear to the entire field.

In a further improved form, and together with a high-performancecomputer system in the mobile device, the analysis could also be carriedout directly on site. The required calculation algorithms could be madeavailable in the form of a smartphone app. Alternatively, a dedicatedcalculating unit (a special processor or special hardware) can beattached to the mobile device or integrated into the mobile device.

Furthermore, it is not absolutely necessary—but is advantageous—to use asmartphone for the digital image. The farmer could also use aconventional digital camera and transmit the digital image of the ear ina different form to the computer for analysis, for example by means ofwired communication technologies or relay stations that use knowncommunication routes such as WLAN, Bluetooth or other comparablecommunication means.

Determination of the natural, non-constant spindle steps of the ear andmultiplication of the determined number of spindle steps by an averagefactor of grains per spindle step allows elegant determination of thenumber of grains per ear. This provides a basis for estimating the fieldyield.

The template matching method used for determining the number of spindlesteps provides, because of the type of digital images of the ear—in theform of the spindle or spindle step view and a potential second digitalimage that is rotated by 90° about the longitudinal axis (flower view)—agood basis for the further image processing and determination steps.

In the following, further embodiments of the suggested method fordetermining a weight of all of the grains of an ear of a grain stalk aredescribed.

According to a embodiment of the method for determining a field yield,the algorithm used for differentiating between image pixels of the earsand other image pixels can be a local binary pattern algorithm. Suchalgorithms are generally known. An example is published in: DC. He andL. Wang, “Texture Unit, Texture Spectrum, and Texture Analysis”, IEEEtransactions on Geoscience and Remote Sensing, vol. 28, pp. 509-512,1990; T. Mäenpää, M. Pietikäinen, and T. Ojala, “Texture classificationby multi-predicate local binary pattern operators”, Proceedings, 15thInternational Conference on Pattern Recognition, Barcelona, Spain,3:951-954, 2000. This provides an effective calculation algorithm thatis directly and easily usable for the proposed method in the form ofexisting program libraries.

According to a further advantageous embodiment of the method, thealgorithm for differentiating between image pixels of the ears and otherimage pixels can be a method for texture image analysis. Such methodsare also generally known, can be adapted according to the requirementsof the method presented and are described for example in: F. Cointault,D. Guerin, J-P. Guillemin & B. Chopinet, “In-field Triticum aestivum earcounting using colour-texture image analysis”, New Zealand Journal ofCrop and Horticultural Science, vol. 36, pp. 117-130, 2008. Thisalgorithm can also be easily adapted to the object position shown here.

According to an additional advantageous embodiment of the method, thealgorithm for differentiating between image pixels of the ears and otherimage pixels can comprise or consist of a brightness difference filter.However, it should be taken into consideration that differences inrecognition may by all means occur depending on illumination, time ofday, color components of sky light, as well as rain, fog, and/orsunlight. For this reason, it can be advantageous to produce the ear orthe field section in all cases using artificial lighting such as anartificial flash.

According to a further advantageous embodiment of the method, thedefined distance between the digital camera and the surface of thecornfield can be determined by means of a spacer between the digitalcamera and an average plane of the ears of the cornfield. The spacer canbe composed of a flexible element-such as e.g. a cord-one end of whichis attached to the digital camera and the other end of which has a colorcontrast ball that is positioned in the average plane of the ears of thecornfield. Because of the color contrast between the color contrast balland the environment (i.e. essentially the ears), the ball is clearlyvisible or recognizable in the digital image by means of patternrecognition. The color contrast ball can for example have a blue orbluish-green color value. In addition to a ball, other regular geometricshapes are also suitable, such as a pyramid, a barrel, a cube, abox-shaped element, or also irregular objects that show good colorcontrast relative to the color values of the ears.

According to an additional embodiment of the method, the defineddistance can be determined by means of a spacer between the digitalcamera and an average plane of the ears of the cornfield. For thispurpose, the digital camera can be attached at a predetermined anglethat is not equal to 90° at one end of the spacer, and the other end ofthe spacer can be positioned on an average plane of the ears of thecornfield. The digital image can be captured when the image plane of thedigital camera is horizontally aligned. For example, the digital cameracan be automatically activated by acceleration or position sensors thatcan be attached to the camera. Here, it is assumed that the averageplane of the ears of the cornfield runs horizontally.

According to the invention, after positioning of the digital camera andcapturing a digital image of a cornfield section, the size of thesurface area of the captured cornfield section is determined(“determining an area of the captured cornfield section from the defineddistance and a viewing angle of the digital camera”). In a further step,the total area of the ears in the digital image is determined from thedigital image (“determining a total area of the ears in the digitalimage compared to the total area of the digital image”). The result, forexample, is that a specified percentage of the pixels of the imagerepresent ears. In a further step, the number of ears in the digitalimage can be determined. For this purpose, one must know the averagearea (how many pixels) taken up by an individual ear. This value can berepresented by the first calibration factor, which is ordinarilyempirically determined. If one divides the total area of the imageattributable to ears by the average size of the area taken up by asingle ear, one obtains the number of ears in the image. If one dividesthe number of ears in the image by the size of the area of the cornfieldin the image section, one obtains the number of ears per unit area ofthe cornfield. If one multiplies the number of ears per unit area of thecornfield by the total area of the field, this gives the number of earsin the entire field. If one multiplies the number of ears in the entirefield by the average grain weight of an ear, this gives the grain weightof the entire field—and thus the yield (“determining a yield of thefield from the total area of the ears in the digital image compared tothe total area of the digital image, the determined area of the capturedcornfield section, the average grain weight of an ear, a total fieldarea and a first calibration factor”). In a preferred embodiment of themethod, the first calibration factor can have at least one dependencywith respect to one of the factors type, growth stage-more particularlyrepresented in the form of the BBCH code-weather, geographic locationand/or fertilization status. Further dependencies are conceivable. Thegeographic location can for example be determined by GPS (globalpositioning system) coordinates. The calibration factor itself can be adirect function of the input variables. Dedicated input values can bestored together with result values in a matrix and be accessed there bymeans of the method.

According to a preferred embodiment of the method, determining the totalarea of the ears in the digital image compared to the total area of thedigital image can further comprise application to the areas of the earsof an area factor, the value of which decreases from the center of thedigital image to its edge. This is advantageous because the ears in thecenter of the image can be better recorded centrally from above, whileears in the edge areas of the digital image-because of the differentviewing angle—can be better recorded from the side and therefore take upa larger section of the image. This effect can be compensated for usingthe area factor.

According to a useful embodiment of the method, the method can compriseproviding a second digital image of an individual ear in a spindle stepview of the ear. The ear in the digital image can be depicted in frontof a reference card as a background.

The method according to this embodiment can further comprise determiningthe length of the ear along the longitudinal axis of the ear byseparating image pixels of the digital image of the ear from thebackground and comparing pixel coordinates at one end of the ear withpixel coordinates of the ear at an opposite end of the ear in alongitudinal direction of the ear by means of image marks on thereference card.

Moreover, the method according to this embodiment can comprisedetermining a number of spindle steps of the ear by means of a templatematching method, determining a number of grains of the ear bymultiplying the determined number of spindle steps by a factor, anddetermining the weight of all of the grains of the ear by multiplyingthe determined number of grains by a second calibration factor.

In this way, the grain weight of an ear can be elegantly determinedwithout requiring weighing out. The optical method allows elegant anddirect determination of the grain weight, either on a smartphone or at aseparate computer center to which the recorded digital image has beentransmitted in the spindle step view. This allows the same technicaldevice-namely the smartphone—to be used both for determining the grainweight and for determining the field yield.

According to an advantageous embodiment of the method, the templatematching method can comprise pixelwise displacement of a selected imagetemplate composed of an average partial area of the ear over the entireear in a longitudinal direction of the ear. Additionally, the method cancomprise respective determination of a respective similarity factor ofthe image template with a respective covered ear section in eachdisplacement position. This allows regular relative maxima to occur withrespect to the similarity factor of an x-y representation. In therepresentation, the x direction of the representation can be the pixelnumber or position, and in the y direction, one can plot a similarityvalue of the respective ear section with the template.

The selected partial area of the ear can take up approx. 15-25% of theear in an average area of the ear. Additionally, this embodiment cancomprise determining the number of spindles from the x-y representation.As the template matching method is a known method from the field ofimage processing, conventional program library functions and modules canbe used. Use of this matching method provides favorable accuracy androbustness against fluctuations in illumination geometry duringdetermination of the spindle steps. This is advantageous because thenumber of spindle steps has a decisive effect on the number of grains ofthe ear. An additional spindle step of the ear can be synonymous with 4additional grains, which can increase the total number of grains of theear by up to 10%. Accordingly, the most accurate detection possible ofthe number of spindle steps can be synonymous with the accuracy of thesuggested method.

According to an additional special embodiment, determining the number ofspindles from the x-y representation in the method can comprisedetermining the number of relative maxima of a similarity value by meansof simple counting. This procedure requires little computing power, butis not the most accurate compared to other methods, because the degreeof similarity decreases toward the ends of the ear and the maxima aretherefore not as pronounced as in the central ear area.

According to a further embodiment, determining the number of spindlesfrom the x-y representation in the method can comprise determining anaverage period length from the distances of the relative maxima of asimilarity value from one another and determining the number of spindlesby dividing the ear length by the period length.

In contrast to the above-described embodiment, the current describedembodiment can show a higher degree of accuracy in determining thenumber of spindle steps. The reason is that the relative maxima in thex-y representation can be more sharply pronounced than in the previousembodiment. This results in higher accuracy in determining the number ofspindle steps of an ear.

In further advantageous examples, the second calibration factor cancomprise at least one dependency with respect to one of the followingfactors: type of ear, growth stage of ear, weather (long-term andshort-term), geographic location and fertilization status. Furtherinfluencing parameters can be taken into consideration at any time.

According to an alternative method for determining the grain weight ofan ear, the method for predicting the yield of a cornfield can comprisea partial method, more particularly a grain weight determination method,for determining a weight of all of the grains of an ear of a grainstalk. This partial method would be an alternative to the partial methodin which a spindle step view of the ear is used. This grain weightdetermination method can comprise providing a digital image of the earin a flower view of the ear in front of a reference card and determiningan area of the flower view of the ear by separating image pixels of thedigital image of the ear from the background by means of a colorhistogram process. Furthermore, this grain weight determination methodcan comprise comparing an area taken up by the ear with image marks onthe reference card. Additionally, the partial method for determininggrain weight of the method for yield prediction can comprise determiningthe weight of all of the grains of the ear by multiplying the determinedarea of the ear by a calibration factor. This calibration factor canshow different dependencies, such as dependency on the type of corn, thegrowth stage, the weather, a fertilization status, a known pestinfestation, etc.

This partial method has the advantage of being easy to use. Thecomputation intensity can be lower than in the partial method fordetermining the grain weight that uses the spindle step view of the ear.This makes this partial method easier to implement directly in a mobiledevice in the field. Alternatively, it would also be possible totransmit the captured digital images to an assessment computing centerand then receive the result with the mobile device in the field. Thispartial method has the further advantage that the digital image of theflower view is easier to prepare than the spindle step view, as the earcomes to rest in a natural position in a flower view. This would make iteaser for the person taking the digital image. It has been found that bymeans of the grain weight determination method presented here,relatively good assessments of the grain weight of an ear can be made.

According to an advantageous embodiment, the system for yield analysisof the cornfield comprises a sending and receiving unit that is adaptedfor transmitting the captured digital image of the cornfield section—oralso the second digital image—to a computer center that comprises thepartial area determination unit, the ear area determination unit and theyield determination module. After calculation is carried out by thedetermination units and the module, the result can be sent back to thesmartphone, the digital camera or the other mobile device and furtherutilized directly in the field.

Moreover, embodiments can take the form of an assigned computer programproduct that can be accessed from a computer-usable or computer-readablemedium. The instructions can cause a computer-such as e.g. a smartphone,a server or a combination of the two—to execute processing stepsaccording to the method presented. For the purpose of this description,the computer-usable or computer-readable medium can be any apparatuscomprising elements for storage, communication, transport ortransmission of the program together with the instruction-processingsystem.

The invention is described in further detail below using examples andfigures. It should be noted that aspects of the invention are describedin the context of various types of examples. Some examples are describedwith respect to process claims, while other examples are described inthe context of device-type claims. Nevertheless, the person havingordinary skill in the art will be able to understand from the above andfollowing descriptions-unless a deviation therefrom has beenindicated—that not only can features of a claim genre be combined withone another, they can also constitute a combination of features thatexceeds the scope of the claim type.

The aspects and further aspects of the present invention presented aboveare derived from the examples and figures, which are described infurther detail below.

These examples serve as possible implementation forms, without beinglimited thereto, and they refer to the following figures:

FIG. 1 shows a block diagram of an embodiment of the method according tothe invention for determining the yield prediction of a cornfield.

FIG. 2 shows positioning of a digital recording device above ears of acornfield.

FIG. 3 shows a means for preferred positioning of the camera above theaverage plane of the ears of the cornfield.

FIG. 4 shows an alternative for reproducible distance positioning of thecamera from the average plane of the ears of the cornfield.

FIG. 5 shows an example of an image of the cornfield corresponding to amethod that was presented in the context of FIG. 4 and FIG. 5.

FIG. 6 shows a block diagram of the partial method for determining thegrain weight of an ear.

FIG. 7 shows a first part of a block diagram of an embodiment of thesuggested method that is closer to implementation.

FIG. 8 shows a second part of the block diagram of the embodiment of thesuggested method that is closer to implementation of FIG. 7.

FIG. 9 shows an abstract representation of an ear and an example of areference card together with an ear lying thereon.

FIG. 9a shows a diagram of an ear and a view of the spindle steps of anear.

FIG. 10 shows an illustrative diagram for determining the ear length.

FIG. 11 shows an illustrative diagram of a cross correlation functionfor determining the number of spindle steps.

FIG. 12 shows a block diagram of a partial system for determining thetotal number of grains of an ear of a grain stalk.

FIG. 13 shows a block diagram of a system for predicting the yield of acornfield.

FIG. 14 shows a block diagram of an example of a computer systemtogether with the system corresponding to FIG. 13 and/or FIG. 12.

In the context of this description, the following conventions, termsand/or expressions may be used:

The term “grain stalk” or “ear of a grain stalk” requires no furtherinterpretation. This can be an ordinary cereal plant that grows in anagricultural field. Typically, the grain can be wheat, rye or barley.

The term “digital image” describes a digital representation of an actualscene that can typically be taken by means of a digital camera. Thedigital image or the digital picture can be composed of pixels havingdiffering color values and thus produce a graphical overall impression.In the method presented here, a digital image of the surface of thecornfield from a bird's eye view and optionally a further digital imageof an individual ear for determining the grain weight of a typical earare taken.

The term “flower view of the ear” describes a view of the ear in whichthe grains are clearly visible. The flower view can also be referred toas a grain view of the ear, because the grains of the ear are the mostclearly visible in this view. In this view, the awns predominantlyextend to the left and right respectively away from the ear. In thisview, the view area of the ear is the largest. In contrast to the flowerview, the term “spindle step view” describes a view of the ear rotatedby 90° along the longitudinal axis of the ear, i.e. a view of the narrowportion of the ear. In this case, one is therefore looking at thenarrower side of the ear or at the awns of the ear if the longitudinalaxis of the ear runs vertically.

In the context of this description, a “reference card” is a flatobject—for example a card having a single color—the color value of whichdiffers sharply from that of the ear. A color that is complementary to atypical color value of the ear—e.g. blue—has been found to beadvantageous.

The term “template matching method” is known to the person havingordinary skill in the art as a method for determining the structure of adigitally represented object. A more detailed description can be foundfor example in S. Kim, J. McNames, “Automatic spike detection based onadaptive template matching for extracellular neural recordings”, Journalof Neuroscience Methods 165, pp. 165-174, 2007.

The term “development stage” describes a stage in the natural life cycleof a plant-hem a grain ear—from sowing until harvest. It has been foundthat using the “BBCH Code” for describing the development stage of aplant is helpful. The abbreviation “BBCH” officially stands for“Biologische Bundesanstalt, Bundessortenamt and Chemische Industrie[Federal Biological Research Centre, Federal Plant Variety Office, andChemical Industry].” The BBCH code describes a phenological developmentstage of plants. The code begins with 00 and ends with 89. For example,a BBCH code of between 10 and 19 describes an early development stage ofa leaf. Beginning with a BBCH code of 60, the flower of the plantappears (up to 69). The next 10 steps respectively describe the fruitdevelopment (70-79), ripening of the seed (80-89) and death (90-99—forannual plants) of the plant.

The term “digital camera” describes a camera that uses a digital storagemedium as a recording medium instead of a photographic film. The digitalimage is first digitized by means of an electronic image converter(image sensor).

The term “color contrast ball” describes an object that comprises aspatial extension that is of the same magnitude as the average length ofthe ears (e.g. a few centimeters in size) and has a weight of themagnitude of approx. 10 to 100 g. Its color is ideally complementary toa dominant color of a surface of a cornfield. Advantageously, this is acomplementary color of a cornfield in a ripe state. For example, thecolor contrast ball can have a blue color value on its surface. It isnot actually necessary for the object to be spherical. Other geometricshapes are also possible. What is important is that the color contrastball be easily distinguishable from pixels of the cornfield by means ofoptical recognition methods.

The term “texture image analysis” describes a method in which thetexture of an object of a digital image is analyzed. In this context,texture is understood to refer to the superficial appearance of theobject or its environment. Examples of methods for texture imageanalysis are described for example in: F. Cointault, D. Guerin, J-P.Guillemin & B. Chopinet, “In-field Triticum aestivum ear counting usingcolour-texture image analysis”, New Zealand Journal of Crop andHorticultural Science, vol. 36, pp. 117-130, 2008.

FIG. 1 shows a block diagram of an example of the method according tothe invention 100 for predicting the yield of a cornfield. The methodcomprises positioning 102 of a digital camera at a defined distanceabove an average plane of ears of a cornfield—i.e. from a bird's eyeview—and capturing 104 of a digital image of a section of the cornfieldwith the positioned digital camera. The average plane of the ears and animage plane of the digital camera should lie parallel to each other.This can be carried out automatically using acceleration or positionsensors of the camera. Automatic activation of the camera can be carriedout precisely when the image plane is horizontally aligned.

Furthermore, the method comprises determination 106 of an area of thecaptured cornfield section from the defined distance and a viewing angleof the digital camera and determination 108 of a total area of the earsin the digital image compared to the total area of the digital image bymeans of an algorithm for differentiating between image pixels of theears and other image pixels that do not belong to the ears. Based onthis, the method comprises determination 10 of a yield of the field fromthe total area of the ears in the digital image compared to the totalarea of the digital image, the determined area of the captured cornfieldsection, the average grain weight of an ear, a total field area and afirst calibration factor. This first calibration factor can depend ontype, growth, weather, geographic location and/or fertilization status.Further dependencies with respect to a pest infestation are alsopossible. For example, the area of the captured cornfield section couldbe 4 m². It could be determined from the analysis of the digital imagethat 20% of the pixels are attributable to ears. Accordingly, for animage size of 1920×1080 pixels, 41,472 pixels would be attributable tocars. Based on empirical tests, it could be determined that an ear underthe conditions in question typically accounts for an average size of 208pixels. In this case, approximately 200 ears would be depicted on thedigital image. For a cornfield section of 4 m², this would correspond toapproximately 50 ears per m² of field area. If the total area of thefield were km², there would be 50 million ears in the entire field. Ifthe average grain weight were 3 g per ear, the total grain weight in thefield would be 150 tons.

FIG. 2 shows positioning of a digital recording device 202 above ears210 of a cornfield. For example, the digital recording device can be anindividual digital camera or a digital camera in a mobile telephone—e.g.a smartphone. The camera 202 captures a defined section of the surfaceof the cornfield. The section is essentially determined by the distanceof the image plane 204 of the camera 202 and the viewing angle α 206 ofthe camera 202. In recording of the digital image, the image plane 204of the camera 202 is advantageously parallel to an average horizontallyrunning plane 208 through the ears 210 of the cornfield.

FIG. 3 shows a means for preferred positioning of the camera 202 abovethe average plane 208 of the ears 210 of the cornfield. The means can bea spacer 302 between the camera 202 and a weight 304. The spacer 302 isattached both to the camera 202 and the weight 304. The weight 304 canfor example consist of a ball. The color of the ball 304 should beclearly distinguishable from the color of the ears 210 or the cornfield.A blue color of the ball 304 is suitable as a complementary color to thecolor of ripe or almost ripe corn (color contrast ball). The spacer 302can be configured in the form of cords or a cord-like structure. Thisensures that because of gravity, the spacer 302 will always beperpendicular to the average plane 208 of the ears 210 of the cornfield.It is also ensured in this manner that the size of the section of thecornfield can be easily calculated based on the length of the spacer302—in the position just described—and the viewing angle of the camera202. In the case of a square digital image, the side length a of thedigital image would be calculated from a=2*L*tan (α/2). Here, L isessentially the length of the spacer 302 and α the viewing angle of thecamera 202.

FIG. 4 shows an alternative for reproducible distance positioning of thecamera 202 from the average plane 208 of the ears 210 of the cornfield.The camera is mounted at the end of a rod 402 at a defined angle. Thisangle is ideally greater than half of the viewing angle of the camera202. If the end 404 of the rod 402 is located at the height of theaverage plane 208 of the ears 210 of the cornfield, the distance302—which is now not defined by a spacer 302—is calculated fromL=Lrod*sin (90−δ), where δ is the angle between the rod 402 and theimage plane 204 of the camera. Based on this known distance of thecamera 202 from the average plane 208 of the ears 210 of the cornfieldand the viewing angle of the camera 202, one can calculate—as shownabove—the area corresponding to the digital image of the camera 202.

Recording of the digital image can be initiated in different ways. Onthe one hand, it is possible to integrate an actuator into the handle404 of the rod 402. In this variant, however, it could be difficult toorient the image plane 204 of the camera 202 so that it is parallel tothe average plane 208 of the ears 210. In a more elegant solution,recording would be automatically initiated as soon as the image plane204 is horizontal after a signal 204 is initiated indicating that thecamera 202 is ready for recording. In this way, it could be ensured thatthe image plane 204 and the average plane 208 of the ears 210 areparallel. An assessment concerning the horizontal orientation of theimage plane 204 can be detected by means of acceleration sensors (orother sensors) of the camera.

FIG. 5 shows an example of an image 500 of a cornfield section. By meansof the methods described above in the context of FIGS. 2 to 4 orcomparable methods, the size of the area of the cornfield correspondingto the digital image can be determined. The ears 502 can be clearly seenin the example image 500. It can also be seen that the ears 502 locatedin the middle of image are recorded at an angle different from that ofthe ears 502 located in the outer areas of the image 500. This arisesfrom simple optical considerations. This effect can be compensated forby means of an area factor that decreases in the direction of the edgesof the digital image 500. Moreover, it can be seen in FIG. 5 thatindividual ears overlap. Accordingly, the number of ears present in theimage cannot be determined for example by determining the number ofcoherent areas of similar brightness values or similar textures.According to the invention, therefore, the total area of the ears in thedigital image is first determined compared to the total area of thedigital image, and in a further step, the number of ears present isdetermined based on a typical size of an ear in a digital image. Theinformation on the typical size of an ear in a digital image is providedby the first calibration factor, which is ordinarily empiricallydetermined.

By determining the area taken up by ears 502 in the image 500, one canextrapolate to find the yield of the entire field.

Advantageously, the grain weight of an ear 502 is taken into account inprojecting the field yield. It is explained in the following figures howsuch a grain weight can also be determined by means of an additionaldigital image of an individual ear.

FIG. 6 shows a block diagram of an example of the expanded method 600for determining the total number of grains of an ear of a grain stalk.The method first comprises the provision 602 of a digital image of theear in a spindle step view of the ear. The ear, in capturing of thedigital image, should be located in front of a reference card as abackground. For practical reasons, the reference card is preferably acolor that is complementary (e.g. blue) to a the typical color of an earof corn (yellowish).

As a further step, the method comprises determination 604 of a length ofthe ear along the longitudinal axis of the ear by separating imagepixels of the digital image of the ear from the background. Thisseparation can be advantageously carried out by means of a colorhistogram process. In this way, a coherent surface of the ear can bedistinguished from the background of the reference card. Additionally,the method comprises in this step comparison 606 of pixel coordinates atone end of the ear with pixel coordinates of the ear at an opposite endof the ear in a longitudinal direction of the ear by means of imagemarks on the reference card. In this manner, by means of a scale locatedon the reference card, the length of the ear can easily be determined.For this purpose, it is necessary only to subtract the corresponding ycoordinates from each other.

It is advantageous if the image of the ear is subjected prior todetermination of its length to a transformation that compensates forperspective distortions and oblique positions.

After this, determination 608 in the method of a number of spindle stepsof the ear by means of a template matching method can be carried out,followed by determining a number of grains of the ear (step 610) bymultiplying the determined number of spindle steps by a factor thatindicates the number of grains per spindle step and for example has avalue of 4.

In a final step of the method, determination 612 is carried out of theweight of all of the grains of the ear by multiplying the determinednumber of grains by a calibration factor. The calibration factor cantake into account numerous variable influencing parameters. Bycontinuously comparing the grain weights determined by the method withthe grain weights determined by weighing out, a continuous andmachine-supported learning process can be implemented within the method.

FIG. 7 shows a first part of a block diagram of an embodiment of thesuggested method that is closer to implementation. A digital image 708of an ear is first received together with a reference card. A geometrycorrection 702 also includes a corner detection 704 of the corners of acolored area on the reference card. After this, transformation 706 ofthe perspectives and image section 710 is carried out so that areasoutside the colored background with the ear lying thereon are ignored.

The image section 710 obtained in this manner is passed on by an eardetection function 712. The actual ear detection takes place by means ofanalysis 714 by a color histogram process in order to differentiatepixels of the ear and the colored background from one another (716foreground/background segmentation). After this, the recognized earobject is masked, 718. In this masked representation, recognized imagepixels of the background can be represented as a logical “0.”

In a subsequent processing block, ear preprocessing 722 is carried out.This can comprise a step of illumination and contrast optimization 724.Next, transformational straightening 726 of the ear and furtherreduction of the image section to be processed can be carried out.Optional awn removal makes it possible to recognize the view of the ear(step 726). Ideally, the view of the ear is a spindle step view. Thefurther processing of the digital image received is carried out based onFIG. 8.

FIG. 8 shows a second part of a block diagram of an embodiment of thesuggested method that is closer to implementation. The actual earanalysis 802 takes place here. For this purpose, a geometry analysis 804is first required, the result of which is determination of the earlength 808. A comparison of the uppermost pixels of the ear with thedepicted and recognized scale or the known size of the colored area ofthe reference card-optionally with the aid of corner marks-allows, inconnection with the distance to the stalk base at the lower end of thestalk base at the lower end of the ear, determination of the length ofthe ear in the longitudinal direction, as shown in FIG. 9.

In the subsequent template matching method 806, a middle selected area810 of the ear 728, in the form displayed at this time, is displacedpixelwise in a vertical direction along the vertical longitudinal axisof the ear 728 above the ear. A respective similarity factor isdetermined that is mathematically determined by the cross correlationfunction 810 between the template and image function. Because of theperiodic pattern of the ear structure, pronounced maxima values arise inthe course of the cross correlation function 810 that lie at a periodicdistance from one another. A result of such a correlation analysis 808is shown in FIG. 10, which yields the period length and thus thedistance of the spindle steps from one another. The ratio of ear length808 to period length provides a highly accurate measurement of half ofthe number of spindle steps, because the periodicity of the spindlesteps is extremely clear and pronounced, with unvarying distance.

After this, on the basis established in this manner, the grain analysis812 is carried out, with determination of the number of grains 816 andthe 1000-grain weight 814. Additionally, this can be followed by yieldcalculation 818 for the entire field, or a partial area thereof, bymeans of a yield formula 820. The 1000 grain weight—also referred to asthousand grain weight (TGW)—is a common calculation value for estimatingyields in an agricultural environment and indicates the weight of 1000grains of a grain batch. It can be calculated from the grain weight ofan ear and the determined number of grains of the ear.

FIG. 9 shows an abstract representation of an ear 708 and an example ofa colored area 902 (not visible in black and white representation) of areference card (that can be larger than the area 902) together with theear 708 lying thereon. The colored area 902 comprises image marks suchas a scale 904, and for example image corner marks 914. The image cornermarks 914 can have various characteristics.

As extensions of the grain 908, awns 906 are symbolically shown that canbe of varying length depending of the type of grain. Moreover, anotherpiece of the stalk 910 is shown, which is important in the methodpresented only for recognition of the lower ear field.

The ear 708 should be oriented on the colored area 902 of the referencecard in such a way that the longitudinal axis 912 of the ear 708 isoriented as close to parallel as possible to a side line of the coloredarea 902. A typical curved form of the ear 708 can be adapted bytransformation of the representation of the ear 708 such that thelongitudinal axis of the ear is actually oriented parallel to a sideline of the colored area 902 of the reference card. The reference cardis typically slightly larger than the colored area 902 containedthereon, the color of which is e.g. blue.

Of course, an actual image of an ear 708 represents a coherent area (forexample as shown in FIG. 7, 720, 728). The type of representation of anear 708 used here is to be understood merely as a representation of theorientation of the ear 708 with respect to the reference card.

FIG. 9a shows a diagram of an ear 708 and a view 926 of the spindlesteps of an ear. The image of the ear 708 clearly shows the differentgrains 916, 918, 920, 922 in the lower area of the ear and the stalk910. Accordingly, the different spindle steps 424 of the ear 708 can berecognized in the more abstract form of the ear on the right side ofFIG. 9 a.

FIG. 10 shows an illustrative diagram 1000 for determining the earlength. Hem, it can be seen that the width of the ear (y axis) isplotted per line of pixels (x axis) pertaining to the ear. Each of theindividual relative maxima—or a group of relative maxima lying closetogether-pertains to a respective spindle step. The number of spindlesteps can be recognized simply by counting the relative maxima or thegroups of relative maxima. The ear length 1002 is determined from thebeginning of the pixels of the ear at approx. line 60 and the end of thepixels of the ear at approx. line 1710 by means of the scale of thereference card or by knowing the width of an individual pixel or line ofpixels.

FIG. 11 shows an illustrative diagram 1100 of a cross correlationfunction for determining the number of spindle steps based on thetemplate matching method. The x axis shows the respective position ofthe template (pattern from the middle of the ear) relative to acorrelation value (similarity value) of the template relative to theentire ear. One can recognize in the center of the diagram—at the pixelvalue of about 525, cf. 1102/max. agreement-a correlation value ofpractically 1. At this location, the template is exactly at its originallocation. Based on the distances of the relative maxima of therepresentation, a period length 1104 can be determined that correspondsto the distance of the individual spindle steps from one another. Basedon the determined length of the ear and the average determined periodlength 1104, the number of spindle steps can also be calculated bydivision and rounding off.

Moreover, an alternative form of the method 600 (i.e. as a replacementfor or supplement to the partial method according to FIGS. 6 to 11) fordetermining the total number of grains of an ear of a grain stalk shouldbe pointed out here: according to this embodiment as well, the processbegins with preparation of a digital image of the ear. In this case, animage of the ear in the flower view—i.e. the view in which the grains ofthe ear are clearly visible—is captured in front of a reference card.This is followed by determining an area of the flower view of the ear byseparating image pixels of the digital image of the ear from thebackground, e.g. by means of a color histogram process, and comparingthe area taken up by the ear by means of image marks on the referencecard. The image marks can be the scale of the reference card orconstitute the known distances of other image marks on the referencecard. After this, the weight of all of the grains of the ear isdetermined by multiplying the determined area of the ear by acalibration factor. It has been found that there is a pronounced directcorrelation between the projection area of the ear in flower view andthe number of grains of the ear. This phenomenon is used here in orderto simply and elegantly determine the grain weight of the ear. Thisalternative method can also be used quite favorably beginning with agrowth stage that is greater than 60 BBHC. However, it also works atlower BBHC values.

FIG. 12 shows a block diagram of a system for determining the totalnumber of grains of an ear of a grain stalk. The system comprises areceiving unit 1202 for receiving a digital image of the ear in a sideview of the ear. If the system for determining the total number ofgrains is integrated into a mobile device, the receiving unit is then adigital camera. In another embodiment, a digital image of the ear iscaptured by a digital camera and transmitted to the receiving unit1202—optionally wirelessly. The ear in the digital image is recorded infront of a reference card as a background. Optionally, the system cancomprise a display unit 1204. Moreover, the system has a measuring unit1206. It is adapted for determining a length of the ear along thelongitudinal axis of the ear by separating image pixels of the digitalimage of the ear from the background. In addition, the measuring unit806 is adapted for comparing pixel coordinates at one end of the earwith pixel coordinates of the ear at an opposite end of the ear in alongitudinal direction of the ear by means of image marks on thereference card.

Additionally, the system comprises a spindle step calculation unit 1208that is adapted for determining a number of spindle steps of the ear bymeans of a template matching method, and a grain number determinationunit 1210 that is adapted for determining a number of grains of the earby multiplying the determined number of spindle steps by a factor.

Finally, a weight determination unit 1212 is also provided in the systemthat is adapted for determining the weight of all of the grains of theear by multiplying the determined number of grains by a calibrationfactor.

As mentioned above, the system can be part of a server system thatreceives the digital image(s) from a digital camera—for example asmartphone. On the other hand, it is also possible-if correspondingcomputing power is available—to integrate the entire system into themobile system, for example into a smartphone or a digital camera.

This system according to FIG. 12 can be integrated with a system 1300for yield analysis of the cornfield shown in FIG. 13. This systemcomprises a digital camera 1302 that is positioned at a defined distanceabove an average plane of ears of a cornfield, wherein the digitalcamera is adapted for capturing a digital image of a section of thecornfield with the positioned digital camera, which can be the same asthat shown in FIG. 12. The average plane of the ears and an image planeof the digital camera should lie parallel to each other. Thedisplay/screen 1304 can also be identical to the display unit 1204 ofFIG. 12.

The system for yield prediction 1300 further comprises a partial areadetermination unit 1306 for determining the area of the capturedcornfield section from the defined distance and a viewing angle of thedigital camera and an ear area determination unit 1308 for determiningthe total area of the ears in the digital image compared to the totalarea of the digital image by means of an algorithm for differentiatingbetween image pixels of the ears and other image pixels that do notbelong to the ears.

Finally, the system 1300 comprises a yield determination module 1310 fordetermining the yield of the field from the total area of the ears inthe digital image compared to the total area of the digital image, thedetermined area of the captured cornfield section, the average grainweight of an ear, a total field area and the first calibration factor.

Embodiments of the invention can be implemented together with virtuallyevery type of computer—in particular also with asmartphone-independently of the platform used for storing and executingthe program code. FIG. 14 shows an example of a computer system 1400that is suitable for implementation of program code relating to theproposed method.

The computer system 1400 is only an example of a suitable computersystem, and it is not intended to represent a limitation of the scope ofuse or functionality of the invention described herein. On the contrary:the computer system 1400 is suitable for implementing any feature or anyfunctionality of the examples described here. The computer system 1400contains components that can work together with numerous other generalor dedicated computer system environments and/or configurations.

Examples of known computer systems, environments and/or configurationsthat can be suitable for working with the computer system 1400 include,without being limited to, tablet computers, notebook computers and/orother mobile computing systems and/or smartphones, as well asmultiprocessor systems, microprocessor-based systems, programmableconsumer electronics or also digital cameras or PDAs (personal digitalassistants).

The computer system 1400 is described here in a general context ofinstructions that can be executed by a computer system. In this case, itcan also be program modules that are executed by the computer system1400. Program modules generally comprise program routines, partialprograms, objects, components, processing and/or decision logic, datastructures, etc., that carry out a specified object or represent aspecified abstract data type.

As mentioned above, the computer system 1400 can be implemented in theform of a “general purpose” computing system. The components of thecomputer system 1400—without being limited hereto-comprise one or aplurality of processing units 1402 (CPUs), a memory system 1404 and asystem bus 1418 that connect different system components-including themain memory 1404 with the processor 1402.

The computer system 1400 also comprises various computer-readable media.Such media comprise all media that are accessible by the computer system1400. This includes both volatile and non-volatile media, which can beeither removable or integrally installed.

The main memory 1404 can also comprise computer-readable media in theform of a volatile memory. This can for example be a random accessmemory (RAM) or also a cache memory. The computer system 1400 canfurther comprise removable and non-removable storage media. The storagesystem 1412 can for example be capable of storing data on anon-removable memory chip. The storage media can be connected to thesystem bus 1406 by one or a plurality of data interfaces. As describedin further detail below, the memory 1404 can comprise at least oneprogram product including a plurality of program modules (at least one)that are configured or can configure the computer system such that thefunctions of the embodiments of the invention can be executed.

A program that comprises a plurality of program modules can be storedfor example in the memory 1404, as can an operating system, one or aplurality of application programs, program modules and/or program data.

The computer system 1400 can further communicate with a plurality ofexternal devices such as a keyboard 1408, a pointer instrument (“mouse”)1410, a display (not shown), etc. These devices can for example becombined in a touch-sensitive screen 1412 (touch screen) in order toallow interaction with the computer system 1400. The computer system1400 can also comprise acoustic input/output devices 1416. Moreover,further connections may also be present in order to allow communicationwith one or a plurality of other data processing devices (modem, networkconnections, etc.). Moreover, such communication can take place viainput/output (I/O) interfaces. Furthermore, the computer system 1400 cancommunicate via one or a plurality of networks-such as a LAN (local areanetwork), a WAN (wide area network) and/or a public (mobile) network(e.g. the Internet) via the adaptor 1414. As shown, the network adapter1414 can communicate with other components of the computer system 1400via the system bus 1418. Moreover, it should be noted—although this isnot shown—that other hardware and/or software components can also beused in connection with the computer system 1400. These include e.g.micro code, device drivers, redundant processing units, etc.

Moreover, the system 1200 for determining a weight of all of the grainsof an ear of a grain stalk or an individual or integrated system 1300for predicting the yield of a cornfield can be connected to the bussystem 1418. In this manner, the computer system or the system 1300 fordetermining the field yield (yield prediction) can receive the digitalimage, carry out determination of the weight of an ear and thus carryout a field yield prediction and send the result back to the mobiledevice with which the digital image(s) were captured. In a particularembodiment, the systems 1200 and or 1300 can also be integrated into amobile computer system (e.g. a high-performance smartphone).

The description of the various embodiments of the present invention isprovided for illustrative purposes. These embodiments are not intendedto limit the scope of the inventive concept. Further modifications andvariations are available to the person having ordinary skill in the artwithout constituting deviations from the core of the present invention.

The present invention can be implemented as a system, a method and/or acomputer program product or a combination thereof. The computer programproduct can comprise a computer-readable storage medium (or simply a“medium”) that contains computer-readable program instructions in orderto cause a processor to implement aspects of the present invention.

This medium can be based on electronic, magnetic or electromagneticwaves, infrared light or semiconductor systems that are also suitablefor transmission. This includes solid-state memory, random access memory(RAM) and read-only memory (ROM). The computer-readable programinstructions described here can be downloaded onto the correspondingcomputer system by a potential service provider via a mobile networkconnection or a stationary network.

The computer-readable program instructions for implementing operationsof the present invention can comprise any kind of machine-dependent ormachine-independent instructions, micro code, firmware, status settingdata, source code or object code written in any desired combination ofone or a plurality of programming languages. The programming languagescan be C++, Java or similar modern programming languages or conventionalprocedural programming languages such as the “C” programming language orsimilar programming languages. The computer-readable programinstructions can be completely executed on the computer system. In otherembodiments, electronic circuits such as e.g. programmable logiccomponents, field-programmable gate arrays (PGAs) or programmable logicarrays (PLA) can execute the instructions using status information inthe computer-readable program instructions to individualize theelectronic circuit(s) in order to carry out aspects of the presentinvention.

Aspects of the present inventions are presented in this document bymeans of flow diagrams and/or block diagrams of methods, apparatuses(systems) and computer program products corresponding to the embodimentsof the invention. It is understood that each block of the flow diagramsand/or block diagrams and combinations of blocks in the flow diagramsand or block diagrams shown can be implemented by computer-readableprogram instructions.

These computer-readable program instructions can be provided to aprocessor of a “general purpose computer” or special computer hardwareor other programmable data processing devices in order to produce amachine such that the instructions executed by the respective processorgenerate means for implementing the functions/actions shown in thecorresponding flow diagram and/or block diagram or blocks thereof. Thesecomputer-readable program instructions can also be stored on acomputer-readable storage medium such that they cause a computer or aprogrammable data processing device to execute the instructions storedon the medium by means of the respective processor, so that aspects oractions of the method described in this document are carried out.

1. A method for predicting the yield of a cornfield, wherein the methodcomprises positioning a digital camera at a defined distance above anaverage plane of ears of a cornfield, capturing a digital image of acornfield section of the cornfield with the positioned digital camera,wherein the average plane of the ears and an image plane of the digitalcamera lie parallel to each other, determining an area of the capturedcornfield section from the defined distance and a viewing angle of thedigital camera, determining a total area of the ears in the digitalimage compared to the total area of the digital image by means of analgorithm for differentiating between image pixels of the ears and otherimage pixels that do not belong to the ears, and determining a yield ofthe field from the total area of the ears in the digital image comparedto the total area of the digital image, the determined area of thecaptured cornfield section, the average grain weight of an ear, a totalfield area and a first calibration factor. 2-17. (canceled)